Scalable video coding system

ABSTRACT

A system for coding video data comprised of one or more frames codes a portion of the video data using a frame-prediction coding technique, and generates residual images based on the video data and the coded video data. The system then codes the residual images using a fine-granular scalability coding technique, and outputs the coded video data and at least one of the coded residual images to a receiver.

This application is a continuation of Ser. No. 09/110,616 filed on Jul.6, 1998

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention is directed to a scalable video coding systemwhich codes video data using both frame-prediction and fine-granularscalable images. The invention has particular utility in connection withvariable-bandwidth networks and computer systems that are able toaccommodate different bit rates, and hence different quality images.

2. Description of the Related Art

Scalable video coding in general refers to coding techniques which areable to provide different levels, or amounts, of data per frame ofvideo. Currently, such techniques are used by lead video codingstandards, such as MPEG-2 and MPEG-4 (i.e., “Motion Picture ExpertsGroup” coding), in order to provide flexibility when outputting codedvideo data.

In the scalable coding techniques currently employed by MPEG-2 andMPEG-4, an encoder codes frames of video data and divides the codedframes into a base layer (“BL”) and an enhancement layer (“EL”).Typically, the base layer comprises a minimum amount of data required todecode the coded video data. The enhancement layer, on the other hand,comprises additional information which enhances (e.g., improves thequality of) the base layer when it is decoded. In operation, the encodertransmits all frames from the base layer to a receiving device, whichcan be a personal computer or the like. However, the encoder onlytransmits frames from the enhancement layer in cases where the receivingdevice has sufficient processing power to handle those additional framesand/or the medium over which the frames are transmitted has sufficientbandwidth.

FIGS. 1 and 2 show “scalability structures” which are currently used inMPEG-2 and MPEG-4 for the base layer and the enhancement layer. Morespecifically, FIG. 1 shows a scalability structure 1 which employsframe-prediction in base layer 2 to generate predicative (or “P”) framesfrom an intra (or “I”) frame or from a preceding P frame. As shown inthe figure, frame-prediction is also used in the enhancement layer togenerate P frames based on frames in the base layer. FIG. 2 showsanother scalability structure 3 which is currently used in MPEG-2 andMPEG-4. In the scalability structure shown in FIG. 2, frame-predictionis again employed to determine P frames in the base layer. Unlikescalability structure 1, however, scalability structure 3 also usesframe-prediction in the enhancement layer to generate bi-directional (or“B”) frames which, in this case, are interpolated from preceding framesin the enhancement layer and contemporaneous frames in the base layer.In general, MPEG-2 and MPEG-4 encoders use frame prediction in themanner set forth above to increase data compression and thus increasecoding efficiency.

Another well-known scalable video coding technique is calledfine-granular scalability coding. Fine-granular scalability coding codesthe same image (e.g., a frame of video) using progressively more dataeach time coding takes place. For example, as shown in FIG. 3, image 4is initially encoded using data sufficient to produce image 5.Thereafter, additional data is coded which is sufficient to produceenhanced images 6, 7 and 8 in succession.

Fine-granular scalability coding has several advantages over theframe-prediction techniques described above. Specifically, becausefine-granular scalability coding can provide a wider range of enhancedimages than frame-prediction techniques, fine-granular scalabilitycoding is generally preferred in environments, such as the Internet,which have a wide range of available bandwidth. For similar reasons,fine-granular scalability coding is also generally preferred whendealing with receiving devices that have varying processing capabilitiesand/or bandwidth. That is, because fine-granular scalability codingproduces a wide range of enhanced images, it is possible to match theappropriate image relatively closely to an amount of availablebandwidth. As a result, in theory, it is possible to obtain the mostamount of data for an image for a given amount of available bandwidth.On the down-side, fine-granular scalability coding does not permit theuse of frame-prediction. As a result, it requires more data than theframe-prediction techniques described above and, consequently, degradescoding efficiency.

Thus, there exists a need for a scalable video coding technique whichincorporates the efficiency of frame-prediction coding and the accuracyof fine-granular scalability coding.

SUMMARY OF THE INVENTION

The present invention addresses the foregoing need by coding a portion(e.g., a base layer) of input video data using a frame-prediction codingtechnique and then coding another portion (e.g., residual images in anenhancement layer) of the video data using fine-granular scalabilitycoding. By coding a base layer using a frame-prediction codingtechnique, the present invention reduces the amount of bits required tocode the video data and thus maintains coding efficiency. By coding theresidual images using fine-granular scalability coding, the presentinvention is able to provide a wide range of residual images, one ormore of which can be selected for transmission based, e.g., on anavailable bandwidth of a receiving device.

Thus, according to one aspect, the present invention is a system (i.e.,a method, an apparatus, and computer-executable process steps) forcoding video data comprised of one or more frames. The system codes aportion (e.g., a base layer) of the video data using a frame-predictioncoding technique, and then generates residual images based on the videodata and the coded video data. Thereafter, the system codes the residualimages using a fine-granular scalability coding technique, and outputsthe coded video data and at least one of the coded residual images to areceiver, such as a variable-bandwidth network or a networked devicethereon.

In preferred embodiments of the invention, the system determines abandwidth of the receiver, and then selects which of the coded residualimages to output based on the bandwidth of the receiver. By doing this,the invention is able to output a coded residual image which is mostappropriate for the available bandwidth.

In other preferred embodiments, the system codes the portion of thevideo data at a plurality of different bit rates so as to producemultiple versions of the coded video data, and generates a plurality ofresidual images for each version of the coded video data. In theseembodiments, the system codes the residual images using a fine-granularscalability coding technique, determines variations in a bandwidth ofthe receiver over time, and then selects which one of the multipleversions and the coded residual images to output based on the variationsin the bandwidth of the receiver.

By way of example, for a receiver bandwidth increasing from B₁ to B₂,where B₁<B₂, the system selects a first version of the coded video dataand successively selects coded residual images corresponding to eachframe of the first version of the coded video data, which are coded atsuccessively higher bit rates. For a receiver bandwidth increasing fromB₂ to B₃, where B₂<B₃, the system selects a second version of the codedvideo data and successively selects coded residual images correspondingto each frame of the second version of the coded video data, which arecoded at successively higher bit rates. Conversely, for a receiverbandwidth decreasing from B₃ to B₂, where B₃>B₂, the system selects afirst version of the coded video data and successively selects codedresidual images corresponding to each frame of the first version of thecoded video data, which are coded at successively lower bit rates.Likewise, for a receiver bandwidth decreasing from B₂ to B₁, whereB₂>B₁, the system selects a second version of the coded video data andsuccessively selects coded residual images corresponding to each frameof the second version of the coded video data, which are coded atsuccessively lower bit rates.

As is clear from the foregoing, by coding a base layer at a plurality ofdifferent bit rates and then selecting versions of the base layer andthe residual images based on a range of available bandwidth, duringdisplay the present invention is able to provide a relatively smoothtransition between different versions of the base layer. That is, inconventional “simulcast” systems (i.e., systems such as this where abase layer has been coded at different bit rates), there is asubstantial jump in image quality at the transition from a first bitrate to a second bit rate. The present invention, however, provides fora smoother transition by selecting and outputting fine-granular codedresidual images between the different versions of the base layer.

According to another aspect, the present invention is a network systemthat includes an encoder which receives input video data and whichoutputs frames of coded video data therefrom, a variable-bandwidthnetwork over which the frames of coded video data are transmitted, adecoder which receives the frames of coded video data from thevariable-bandwidth network and which decodes the coded video data, and adisplay which displays the decoded video data. The encoder includes aprocessor and a memory which stores computer-executable process steps.The processor executes process steps stored in the memory so as toproduce the frames of coded video data by (i) coding a base layer fromthe input video data using a frame-prediction coding technique, (ii)coding an enhancement layer from the input video data using afine-granular scalability coding technique, (iii) determining abandwidth of the variable-bandwidth network, and (iv) selecting, foroutput, the base layer and, in a case that the bandwidth of thevariable-bandwidth network is greater than a predetermined value, aportion of the enhancement layer.

According to still another aspect, the present invention is a system fordecoding video data comprised of an enhancement layer bitstream and abase layer bitstream, where the base layer bitstream is coded using aframe-prediction coding technique and the enhancement layer bitstream isencoded using a fine-granular scalability coding technique. The systemreceives the coded video data, decodes the base layer bitstream using aframe-prediction decoder, and decodes the enhancement layer bitstreamusing a fine-granular scalability decoder. Thereafter, the systemcombines (e.g., adds) decoded video data from the base layer bitstreamand from the enhancement layer bitstream to form a video image.

According to still another aspect, the present invention is a system forcoding video data and outputting coded video data to a plurality ofreceivers. The system codes a first portion of the video data using aframe-prediction coding technique to produce a first bitstream, and thencodes a second portion of the video data using a fine-granularscalability coding technique to produce a second bitstream. The firstbitstream is output to the plurality of receivers, whereafter the secondbitstream is divided into two or more sub-streams. Finally, the two ormore sub-streams are output to the plurality of receivers.

By virtue of the foregoing aspect of the invention, it is possible tomulticast video data to a plurality of receivers. In other words, it ispossible to broadcast coded data to the receivers at multiplebandwidths. These receivers may then accept only those bandwidths thatthey are able to process and/or receive. Thus, each receiver is able toreceive and process as much data as it can handle, thereby resulting inmore accurate image reproduction thereby.

This brief summary has been provided so that the nature of the inventionmay be understood quickly. A more complete understanding of theinvention can be obtained by reference to the following detaileddescription of the preferred embodiments thereof in connection with theattached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 depicts a scalability structure used with a conventionalframe-prediction-type scalable coding technique.

FIG. 2 depicts an alternative scalability structure used with aconventional frame-prediction-type scalable coding technique.

FIG. 3 depicts images generated using a fine-granular scalabilitycoding/decoding technique.

FIG. 4 depicts a computer system on which the present invention may beimplemented.

FIG. 5 depicts the architecture of a personal computer in the computersystem shown in FIG. 4.

FIG. 6 is a functional block diagram showing elements of the first andsecond embodiments of the present invention.

FIG. 7 is a flow diagram describing the scalability coding technique ofthe present invention.

FIG. 8 shows a scalability structure generated by the present invention.

FIG. 9 is a block diagram of a decoder in accordance with the presentinvention.

FIG. 10 is a graph depicting image quality versus bit-rate for simulcastbitstreams generated by the second embodiment of the present invention.

FIG. 11 is a functional block diagram showing elements of the thirdembodiment of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

FIG. 4 shows a representative embodiment of a computer system 9 on whichthe present invention may be implemented. As shown in FIG. 4, personalcomputer (“PC”) 10 includes network connection 11 for interfacing to anetwork, such as a variable-bandwidth network or the Internet, andfax/modem connection 12 for interfacing with other remote sources suchas a video camera (not shown). PC 10 also includes display screen 14 fordisplaying information (including video data) to a user, keyboard 15 forinputting text and user commands, mouse 13 for positioning a cursor ondisplay screen 14 and for inputting user commands, disk drive 16 forreading from and writing to floppy disks installed therein, and CD-ROMdrive 17 for accessing information stored on CD-ROM. PC 10 may also haveone or more peripheral devices attached thereto, such as a scanner (notshown) for inputting document text images, graphics images, or the like,and printer 19 for outputting images, text, or the like.

FIG. 5 shows the internal structure of PC 10. As shown in FIG. 5, PC 10includes memory 20, which comprises a computer-readable medium such as acomputer hard disk. Memory 20 stores data 23, applications 25, printdriver 24, and operating system 26. In preferred embodiments of theinvention, operating system 26 is a windowing operating system, such asMicrosoft® Windows95; although the invention may be used with otheroperating systems as well. Among the applications stored in memory 20are scalable video coder 21 and scalable video decoder 22. Scalablevideo coder 21 performs scalable video data encoding in the manner setforth in detail below, and scalable video decoder 22 decodes video datawhich has been coded in the manner prescribed by scalable video coder21. The operation of these applications is described in detail below.

Also included in PC 10 are display interface 29, keyboard interface 30,mouse interface 31, disk drive interface 32, CD-ROM drive interface 34,computer bus 36, RAM 37, processor 38, and printer interface 40.Processor 38 preferably comprises a microprocessor or the like forexecuting applications, such those noted above, out of RAM 37. Suchapplications, including scalable video coder 21 and scalable videodecoder 22, may be stored in memory 20 (as noted above) or,alternatively, on a floppy disk in disk drive 16 or a CD-ROM in CD-ROMdrive 17. Processor 38 accesses applications (or other data) stored on afloppy disk via disk drive interface 32 and accesses applications (orother data) stored on a CD-ROM via CD-ROM drive interface 34.

Application execution and other tasks of PC 4 may be initiated usingkeyboard 15 or mouse 13, commands from which are transmitted toprocessor 38 via keyboard interface 30 and mouse interface 31,respectively. Output results from applications running on PC 10 may beprocessed by display interface 29 and then displayed to a user ondisplay 14 or, alternatively, output via network connection 11. Forexample, input video data which has been coded by scalable video coder21 is typically output via network connection 11. On the other hand,coded video data which has been received from, e.g., a variablebandwidth-network is decoded by scalable video decoder 22 and thendisplayed on display 14. To this end, display interface 29 preferablycomprises a display processor for forming video images based on decodedvideo data provided by processor 38 over computer bus 36, and foroutputting those images to display 14. Output results from otherapplications, such as word processing programs, running on PC 10 may beprovided to printer 19 via printer interface 40. Processor 38 executesprint driver 24 so as to perform appropriate formatting of such printjobs prior to their transmission to printer 19.

First Embodiment

Turning to scalable video coder 21, this module comprisescomputer-executable process steps which code video data comprised of oneor more successive frames. In brief, these process steps code a portionof the video data using a frame-prediction coding technique, generateresidual images based on the video data and the coded video data, andcode the residual images using a fine-granular scalability codingtechnique. The steps then output the coded video data and at least oneof the coded residual images to a receiver which, generally speaking,can comprise a network (variable-bandwidth or otherwise), a PC, or othervideo-supporting networkable devices including, but not limited to,digital televisions/settop boxes and video conferencing equipment.

FIG. 6 is a block diagram depicting a video source 42, avariable-bandwidth network 43, and modules used to effect the foregoingprocess steps. FIG. 7 is a flow diagram which explains the functionalityof the modules shown in FIG. 6. To begin, in step S701 original uncodedvideo data is input into the present invention. This video data may beinput via network connection 11, fax/modem connection 12, or, as shownin FIG. 6, via a video source. For the purposes of the presentinvention, video source 42 can comprise any type of video capturingdevice, an example of which is a digital video camera. As shown in FIG.6, video data from the video source is input to both BL encoder 44 andresidual image computation block 45. The reason for this is apparentbelow.

Next, step S702 codes a portion (i.e., a base layer, or BL) of theoriginal video data using a standard frame-prediction coding technique.Step S702 is performed by BL encoder 44, which, in preferred embodimentsof the invention, is an MPEG-1, an MPEG-2 or an MPEG-4 encoder. Ageneral overview of the MPEG standard is provided in “MPEG: A VideoCompression Standard For Multimedia Applications”, by Didier LeGall,Communications of the ACM, Vol. 34, No. 4 (April 1991). BL encoder 44compresses the video data at a predetermined bit-rate, R_(BL). Inpreferred embodiments of the invention, R_(BL) is determined bycalculation block 48 based on a current bandwidth of a receiver, such asvariable-bandwidth network 43 (or, e.g., a computer system havingvariable processing capabilities).

More specifically, calculation block 48 measures a minimum bit-rate(“R_(MIN)”), a maximum bit-rate (“R_(MAX)”), and a current availablebandwidth (“R”) of variable-bandwidth network 43. Calculation block 48then sets R_(BL) to a value between R_(MIN) and R. In most cases,calculation block 48 sets R_(BL) to R_(MIN), so as to ensure that, evenat its lowest bandwidths, variable-bandwidth network 43 will be able toaccommodate coded video data output by the present invention. This isespecially true in cases where base layer encoding takes place off-line.

FIG. 8 shows an example of a scalability structure which is generated bythe present invention. As shown in FIG. 8, this scalability structureincludes both a base layer (“BL”) and an enhancement layer (“EL”). Baselayer 47 includes frames, such as frame 49. These frames are compressedat a bit-rate of R_(BL) by BL encoder 44. Enhancement layer 50, however,includes fine-granular coded images corresponding to contemporaneousframes in the base layer. The following describes how the inventiongenerates enhancement layer 50.

More specifically, step S703 generates residual images 51 based on theoriginal video data input from video source 42 and based on coded videodata (i.e. the base layer) provided by BL encoder 44. In the blockdiagram shown in FIG. 6, step S703 is performed by residual imagecomputation block 45. In operation, residual image computation block 45receives coded video data from BL encoder 44 and then decodes that codedvideo data. Thereafter, residual images 51 are generated based on adifference between pixels in this decoded video data and pixels in theoriginal video data. Generally speaking, the residual images correspondto the difference between frames in the base layer (which comprises theminimum number of frames and/or the minimum amount of data required by adecoder to decode a video signal) and frames in the original video data.

Residual image computation block 45 may use one or more of variety ofdifferent methods to generate residual images 51. For example, in oneembodiment of the invention, a simple pixel-by-pixel subtraction isperformed between frames in the base layer and frames in the originalvideo data. The resulting difference between these two sets of frames(i.e., the residual images) includes differences in the frames'resolutions. In cases where the base layer does not include entireframes of the original video data, the residual images include thesemissing frames.

In another embodiment of the invention, residual image computation block45 generates residual images 51 by first filtering the decoded videodata and then determining a difference between this filtered video dataand the original video data. This technique has the advantage ofremoving unwanted noise and the like from the decoded video data caused,e.g., by the coding and decoding processes. In preferred embodiments ofthe invention, a deblocking filter is used to filter the decoded videodata; although the invention is not limited to the use of this type offilter.

In still another embodiment of the invention, residual image computationblock 45 generates residual images 51 by filtering both the decodedvideo and the original video data, and then determining a differencebetween both of these types of filtered data. In this embodiment, thesame type of filter (e.g., a deblocking filter) may be applied to boththe original video data and the decoded video data. Alternatively,different types of filters may be applied to the original video data andto the decoded video data.

In general, when filtering is used to generate residual images 51, adecoder for receiving video data that has been coded in accordance withthe present invention should be “in synch” with the type of filteringused thereby, meaning that substantially the same type of filteringshould be applied at the decoder in order to compensate for the effectsof filtering. For example, if residual images 51 are coded based onfiltered decoded video data, that same filtering should be applied tothe residual images during decoding thereof.

Returning to FIG. 7, after step S703, processing proceeds to step S704.Step S704 codes the residual images using an embedded fine-granularscalability coding technique, as shown in the enhancement layer of thescalability structure of FIG. 8. In the embodiment of the inventionshown in FIG. 6, this step is performed by fine-granular scalable ELencoder 54. EL encoder 54 codes residual images 51 at a bit-rate ofR_(MAX)-R_(BL) (i.e., the difference between the base layer bandwidthand maximum bandwidth of network 43) using a fine-granular codingtechnique. At this point, it is noted that, since a fine-granularscaling technique is used to code frames for the enhancement layer,frame prediction is not employed therein.

As shown in FIG. 6, values for R_(MAX) and R_(BL) are provided to ELencoder 54 by calculation block 48. Any of a variety of well-knownfine-granular coding techniques may be used by EL encoder 54. Examplesof these include an embedded discrete cosine transform (“DCT”) techniqueand a scalable matching pursuit (“MP”) technique. Preferred embodimentsof the invention, however, use one of the family of wavelet transforms(e.g., zero tree wavelet transforms) to effect enhancement layer coding.For example, the preferred embodiment of the invention uses thestill-image coding technique provided in MPEG-4 to perform fine-granularscalability coding. This approach codes images as whole using wavelettransforms.

Regardless of what type of fine-granular scalability coding is used byEL encoder 54, an EL bitstream is output therefrom which has a bit-rateof R_(MAX)-R_(BL). This EL bitstream comprises a plurality of embeddedfine-granular scalable images, meaning that the bitstream is comprisedof an initial coarse image and one or more enhancements thereto. Forexample, the EL bitstream may include a coarse image comprised of apredetermined number of bits (e.g., the first 100 bits) in thebitstream; an enhancement image comprising the coarse image and the nextpredetermined number of bits (e.g., the next 100 bits) in the bitstream;a further enhancement image comprising the coarse image, the enhancementimage, and the next predetermined number of bits (e.g., the next 100bits) in the bitstream; and so on. The number of bits used to enhancethese images (100 bits in this example) is referred to as the image'sgranularity.

At this point, it is noted that the present invention is not limited tousing 100 bit granularity, or even to using the same number of bits toenhance the image. In fact, the granularity used by the invention canvary and, in preferred embodiments, can reach down to the byte level oreven to the single bit level wherein single bits are used to enhance animage.

As shown in FIG. 6, the EL bitstream is provided to real-time scalablevideo rate controller 55 which performs, in real-time, steps S705 andS706 shown in FIG. 7. In step S705, controller 55 receives R_(BL),R_(MAX) and R from calculation block 48, and then selects, for eachframe in the base layer, one or more of the coded residual images inenhancement layer 50 (see FIG. 8) based on these values. In particular,controller 55 selects image(s) from the enhancement layer which have abandwidth that substantially corresponds to R-R_(BL), i.e., thedifference between the actual bandwidth of network 43 and the bandwidthof the base layer. Controller 55 selects these images by transmittingimages from the EL bitstream (e.g., a coarse image and/or imageenhancements) having a bandwidth that corresponds to R-R_(BL), andblocking transmission of those image enhancements which fall outside ofthat range. By implementing the invention using a relatively finegranularity, such as single-bit granularity, the invention is able tofill substantially all of the bandwidth between R and R_(BL). In thesecases, the invention is able to provide substantially the maximum amountof video data for the given amount of available bandwidth. Of course, incases where the receiver can handle only coded images from the baselayer, controller 55 will not transmit any fine-granular scalable imagesfrom the enhancement layer.

Assuming, however, that these images are to be transmitted, once theappropriate fine-granular scalable images (i.e., coded residual images)have been selected by controller 55, processing proceeds to step S706.In step S706, controller 55 outputs the base layer and the fine-granularscalable images selected in step S705. As shown in FIG. 6, the imagesare output to variable-bandwidth network 43 as a BL stream and an ELstream.

A decoder, a functional block diagram for which is shown in FIG. 9, thenreceives these coded bitstreams and decodes the data therein. Decoder 57may comprise a PC, such as that shown in FIG. 4 or, alternatively, anyof the other receivers mentioned above. As shown in the figure, decoder57 includes a scalable video decoder module 58 which is executed by aprocessor therein. This scalable video decoder module is comprised of afine-granular scalable EL decoding module 59 for decoding data in the ELbitstream and a frame-prediction BL decoding module 60 for decodingframes in the BL bitstream. In preferred embodiments of the presentinvention, BL decoding module 60 comprises an MPEG-1, MPEG-2 or MPEG-4decoding module. Due to the fine granularity of the EL bitstream, the ELdecoder can decode any appropriate portion of the EL bitstream limited,e.g., by decoder processing constraints or the like. Once the respectivedecoding modules have decoded the streams of video data, framestherefrom are added and reordered, if necessary, by processing block 61.These frames may then be displayed to a user.

Second Embodiment

The second embodiment of the present invention generates a scalabilitystructure like that shown in FIG. 8 for each of a plurality of“simulcast” bitstreams. Briefly, in the second embodiment of the presentinvention, scalable video coder 21 includes computer-executable processsteps to code a portion (e.g., the base layer) of input video data at aplurality of different bit rates so as to produce multiple versions ofcoded video data, to generate a plurality of residual images for eachversion of the coded video data, to code the plurality of residualimages for each version of the coded video data using a fine-granularscalability coding technique, and then to output one version (e.g., onebase layer) of the coded video data together with one or more codedresidual images therefor.

More specifically, in this embodiment of the invention, BL encoder 44codes the base layer at a plurality of different bit rates R_(B1),R_(B2), R_(B3) . . . R_(BN), where

 R _(MIN) <R _(B1) <R _(B2) <R _(B3) . . . <R _(BN) <R _(MAX).

For each of these resulting simulcast coded bitstreams, residual imagecomputation block 45 generates residual images in the manner describedabove. Thereafter, EL encoder 54 generates corresponding fine-granularcoded images for each set of residual images. These fine-granular codedimages have bit-rates of R_(E1), RE₂, R_(E3) . . . R_(EN), which aredetermined in substantially the same manner as those of the EL bitstreamof the first embodiment. That is, $\begin{matrix}\begin{matrix}{R_{E1} = {R_{{E1}\quad {MAX}} - R_{B1}}} \\{R_{E2} = {R_{{E2}\quad {MAX}} - R_{B2}}} \\\vdots \\{R_{E{({N - 1})}} = {R_{{E{({N - 1})}}\quad {MAX}} - R_{B{({N - 1})}}}} \\{{R_{N} = {R_{MAX} - R_{BN}}},}\end{matrix} & (1)\end{matrix}$

where R_(EM)∈[R_(BM), R_(MAX)] and M∈[1,N]. In a case that the maximumEL bit-rate for a particular BL bitstream is set as the minimum bit-rateof a next simulcast BL bitstream, equations (1) reduce to$\begin{matrix}\begin{matrix}{R_{E1} = {R_{B2} - R_{B1}}} \\{R_{E2} = {R_{B3} - R_{B2}}} \\\vdots \\{R_{E{({N - 1})}} = {R_{BN} - R_{B{({N - 1})}}}} \\{R_{N} = {R_{MAX} - {R_{BN}.}}}\end{matrix} & (2)\end{matrix}$

FIG. 10 is an example of a graph of image quality versus bit-rate whichexplains the case corresponding to equations (2). More specifically, asshown in FIG. 10, the invention initially selects a scalabilitystructure having a base layer with a bit-rate R_(B1) (which, in thiscase is R_(MIN)). The invention then monitors parameters ofvariable-bandwidth network 43 via calculation block 48, and determines anew bandwidth R therefor periodically. As the bandwidth ofvariable-bandwidth network 43 increases over time, controller 55 selectsprogressively more detailed fine-granular coded residual images for eachframe of the selected scalability structure/base layer, and outputsthose images to the receiver. The receiver then provides those image toa display, such as display 14 above, thereby leading to the progressiveincrease in image quality shown by line 64 in FIG. 10. However, usingthe scalability structure for R_(B1), it is only possible to provide alimited increase in image quality, as shown by dotted line 65 in FIG.10.

Accordingly, once the bandwidth R of variable bandwidth network 43reaches a predetermined level (which may be pre-set in controller 55),the scalability structure for bit-rate R_(B2) is selected. As was thecase above, the invention then continues to monitor variable-bandwidthnetwork 43 via calculation block 48, and to re-calculate the bandwidththereof over time. As the bandwidth of variable-bandwidth network 43increases, controller 55 selects progressively more detailedfine-granular coded residual images for each frame of the selectedscalability structure/base layer, and outputs those images to thereceiver. The receiver then provides those image to a display, such asdisplay 14 above, thereby leading to the further progressive increase inimage quality shown by line 66 in FIG. 10. A process similar to this isperformed up to R_(MAX).

By virtue of the foregoing process, this embodiment of the invention isable to use simulcast bitstreams to provide an overall increase imagequality without large “jumps” at transition points R_(B2) and R_(B3).That is, conventional systems which use simulcast bitstreams to increaseimage quality have a large “jump” at each transition point between twosimulcast bitstreams. This results in an abrupt transition in thedisplayed image. In contrast, because the present invention usesfine-granular images between the transition points, the invention isable to provide a gradual transition between bitstreams, along with acontinuous increase in image quality over time.

Of course, the converse of the foregoing occurs for variable-bandwidthnetworks that have decreasing bandwidth. That is, for a receiverbandwidth decreasing from B₃ to B₂, where B₃>B₂, the invention selects afirst base layer and successively selects fine-granular coded residualimages corresponding to each frame of the first base layer that arecoded at successively lower bit rates. As the bandwidth decreases fromB₂ to B₁, where B₂>B₁, the invention selects a second base layer andsuccessively selects fine-granular coded residual images correspondingto each frame of the second base layer that are coded at successivelylower bit rates. This results in a relatively smooth decrease in imagequality, as opposed to an abrupt transition. Of course, relativelysmooth transitions are also achieved by the present invention forvariable-bandwidth networks that have neither continuously increasingnor continuously decrease bandwidths, but rather have fluctuating oroscillating bandwidths. Such is also the case for computer systems orthe like which have varying processing capabilities

At this point, it is noted that although the first two embodiments ofthe present invention have been described with respect to avariable-bandwidth network, these embodiments can be used outside of anetwork context. That is, rather than measuring network bandwidth, theinvention may measure the processing capabilities of a receiving device(e.g., a PC) and then vary coding accordingly.

Third Embodiment

FIG. 11 depicts a third embodiment of the present invention. In brief,this embodiment is a method and corresponding apparatus and processsteps for coding video data and for multicasting coded video data to aplurality of receivers. In this embodiment, scalable video coder 21codes a first portion of the video data (e.g., the base layer) using aframe-prediction coding technique to produce a first bitstream (e.g.,the BL bitstream), and then codes a second portion of the video data(e.g., the enhancement layer) using a fine-granular scalability codingtechnique to produce a second bitstream (e.g., the EL bitstream).Thereafter, the first bitstream is output to one or more of theplurality of receivers, and the second bitstream is divided into two ormore sub-streams These two or more sub-streams are then also output tothe plurality of receivers.

As shown in FIG. 11, the third embodiment of the invention includesvideo source 70, BL encoder 71, residual image computation block 72, andEL encoder 73. These features are identical to those described abovewith respect to the first embodiment. Accordingly, detailed descriptionsthereof are omitted herein for the sake of brevity. As shown in FIG. 11,the third embodiment also includes multicast rate controller 74 andcalculation block 75. Detailed descriptions of these features of theinvention are as follows.

Calculation block 75 is similar to calculation block 48 described abovein that it determines R_(MIN), R_(MAX) and R_(BL). In this embodiment,however, R_(MIN) comprises the minimum bandwidth among plural receivers(e.g., PCs) on network 76 and R_(MAX) comprises the maximum bandwidthamong the plural receivers on network 76. As above, calculation block 75sets R_(BL) to a value between R_(MIN) and R_(MAX), and usually toR_(MIN) so as to ensure that even the lowest bandwidth receiver will beable to process coded video data output by the present invention. Asshown in FIG. 11, in this embodiment of the invention, calculation block75 also determines bandwidths R₁, R₂ . . . R_(N) for correspondingcategories of receivers 1, 2 . . . N (not shown) on network 76. This maybe done by monitoring the network for traffic to and from thesereceivers and/or issuing status inquiries to the respective receivers.Thereafter, these values for R₁, R₂ . . . R_(N) are provided tomulticast rate controller 74.

Multicast rate controller 74 uses R₁, R₂ . . . R_(N) to divide the ELbitstream into sub-streams ranging from 0 bits to R_(N) bits. That is,as shown in FIG. 11, multicast rate controller 74 divides the ELbitstream into sub-streams having bandwidths of:

0→R ₁ R ₁ →R ₂ R _(N−1) →R _(N),  (3)

where R_(N) is less than or equal to R_(MAX)-R_(BL). Each of thesesub-streams corresponds to embedded fine-granular coded residual images.Specifically, the 0 to R₁ bitstream comprises a coarse image; the R₁ toR₂ sub-stream comprises an enhancement to the coarse image; and so on.The sub-streams described in expression (3) above are then output toreceivers on network 76, together with the BL bitstream. These receiverswill then accept the BL bitstream and one, some, all, or none of thesesub-streams, depending upon the processing capabilities of the receiverand/or the network. Decoders, such as that shown in FIG. 9, at thesereceivers may then be used to decode the bitstreams.

Of course, those skilled in the art will realize that it is alsopossible to combine the second and third embodiments of the invention soas to produce an encoder which multicasts sub-streams for a plurality ofsimulcast BL bitstreams. In addition, although this embodiment has beendescribed with respect to networked receivers, it is noted that theembodiment can be used with non-networked receivers as well. Theinvention can also be used to provide coded data to a plurality ofvariable-bandwidth networks connected, e.g., to a single PC or the likevia plural network connections.

Likewise, although the three embodiments of the invention describedherein are preferably implemented as computer code, all or some of thecomponents shown in FIGS. 6 and 11 can be implemented using discretehardware elements and/or logic circuits. The same is true for thedecoder shown in FIG. 9. Thus, for example, calculation blocks 48 and 75can comprise a workstation, PC or other operator-driven device forinputting and selecting required control and command parameters. Lastly,while the encoding and decoding techniques of the present invention havebeen described in a PC environment, these techniques can be used in anytype of video devices including, but not limited to, digitaltelevisions/settop boxes, video conferencing equipment, and the like.

In this regard, the present invention has been described with respect toparticular illustrative embodiments. It is to be understood that theinvention is not limited to the above-described embodiments andmodifications thereto, and that various changes and modifications may bemade by those of ordinary skill in the art without departing from thespirit and scope of the appended claims.

What is claimed is:
 1. A method of coding video data comprised of one ormore frames, the method comprising: a first coding step for producingcoded video data by coding a portion of the video data using aframe-prediction coding technique; a generating step for generatingresidual images based on the video data and the coded video data; asecond coding step for producing coded residual images by coding theresidual images using a fine-granular scalability coding technique; andan outputting step for outputting the coded video data and one or moreof the coded residual images to a receiver.
 2. A method according toclaim 1, further comprising the steps of: determining a bandwidth of thereceiver; and selecting which of the coded residual images to output inthe outputting step based on the bandwidth of the receiver.
 3. A methodaccording to claim 2, wherein the coded residual images comprise, foreach frame of the coded video data, a plurality of differentfine-granular scalable images each coded at a different bit rate; andwherein the selecting step selects, for each frame of the coded videodata, a coded residual image having a highest bit rate that can beaccommodated by the bandwidth of the receiver.
 4. A method according toclaim 3, wherein the selecting step is performed in real-time by areal-time scalable video rate controller.
 5. A method according to claim1, wherein the first coding step codes the portion of the video datausing one of MPEG-1, MPEG-2 and MPEG-4.
 6. A method according to claim1, wherein the generating step comprises the steps of: decoding thecoded video data to produce decoded video data; and determining theresidual images by determining a difference between pixels in the videodata and pixels in the decoded video data.
 7. A method according toclaim 1, wherein the generating step comprises the steps of: decodingthe coded video data to produce decoded video data; filtering thedecoded video data to produce filtered video data; and determining theresidual images by determining a difference between pixels in the videodata and pixels in the filtered video data.
 8. A method according toclaim 7, wherein the filtering step is performed using a deblockingfilter.
 9. A method according to claim 1, wherein the generating stepcomprises the steps of: filtering the video data to produce firstfiltered video data; decoding the coded video data to produce decodedvideo data; filtering the decoded video data to produce second filteredvideo data; and determining the residual images by determining adifference between pixels in the first filtered video data and pixels inthe second filtered video data.
 10. A method according to claim 1,wherein the fine-granular coding technique comprises a member of thewavelet transform family of coding techniques.
 11. A method according toclaim 1, wherein the fine-granular coding technique comprises anembedded discrete cosine transform (“DCT”) coding technique.
 12. Amethod according to claim 1, wherein the fine-granular coding techniquecomprises a scalable matching pursuit (“MP”) coding technique.
 13. Amethod according to claim 1, wherein the receiver comprises avariable-bandwidth network.
 14. A method according to claim 1, whereinthe first coding step comprises coding the portion of the video data ata plurality of different bit rates so as to produce multiple versions ofthe coded video data; wherein the generating step comprises generating aplurality of residual images for each version of the coded video data;wherein the second coding step comprises coding the plurality ofresidual images for each version of the coded video data using afine-granular scalability coding technique; and wherein the outputtingstep comprises outputting one version of the coded video data togetherwith at least one corresponding coded residual image therefor.
 15. Amethod according to claim 14, wherein the outputting step comprises thesteps of: determining variations in a bandwidth of the receiver overtime; and selecting which one of the multiple versions of the codedvideo data and which of the coded residual images to output over timebased on the variations in the bandwidth of the receiver.
 16. Anapparatus for coding video data comprised of one or more frames, theapparatus comprising: a memory which stores computer-executable processsteps; and a processor which executes the process steps stored in thememory so as (i) to produce coded video data by coding a portion of thevideo data using a frame-prediction coding technique, (ii) to generateresidual images based on the video data and the coded video data, (iii)to produce coded residual images by coding the residual images using afine-granular scalability coding technique, and (iv) to output the codedvideo data and at least one of the coded residual images to a receiver.17. An apparatus to claim 16, wherein the processor executes processsteps stored in the memory so as (i) to determine a bandwidth of thereceiver, and (ii) to select which of the coded residual images tooutput in the outputting step based on the bandwidth of the receiver.18. An apparatus according to claim 17, wherein the coded residualimages comprise, for each frame of the coded video data, a plurality ofdifferent fine-granular scalable images each coded at a different bitrate; and wherein the processor selects, for each frame of the codedvideo data, a coded residual image having a highest bit rate that can beaccommodated by the bandwidth of the receiver.
 19. An apparatusaccording to claim 18, wherein the processor executes a real-timescalable video rate controller to perform the outputting.
 20. Anapparatus according to claim 16, wherein the processor codes the portionof the video data using one of MPEG-1, MPEG-2 and MPEG-4.
 21. Anapparatus according to claim 16, wherein the processor generates theresidual images by (i) decoding the coded video data to produce decodedvideo data, and (ii) determining the residual images by determining adifference between pixels in the video data and pixels in the decodedvideo data.
 22. An apparatus according to claim 16, wherein theprocessor generates the residual images by (i) decoding the coded videodata to produce decoded video data, (ii) filtering the decoded videodata to produce filtered video data, and (iii) determining the residualimages by determining a difference between pixels in the filtered videodata and pixels in the video data.
 23. An apparatus according to claim22, wherein the processor filters the decoded video data using adeblocking filter.
 24. An apparatus according to claim 16, wherein theprocessor generates the residual images by (i) filtering the video datato produce first filtered video data, (ii) decoding the coded video datato produce decoded video data, (iii) filtering the decoded video data toproduce second filtered video data, and (iv) determining the residualimages by determining a difference between pixels in the first filteredvideo data and pixels in the second filtered video data.
 25. A methodaccording to claim 16, wherein the fine-granular coding techniquecomprises a member of the wavelet transform family of coding techniques.26. A method according to claim 16, wherein the fine-granular codingtechnique comprises an embedded discrete cosine transform (“DCT”) codingtechnique.
 27. A method according to claim 16, wherein the fine-granularcoding technique comprises a scalable matching pursuit (“MP”) codingtechnique.
 28. An apparatus according to claim 16, wherein the receivercomprises a variable-bandwidth network.
 29. An apparatus according toclaim 16, wherein the processor (i) codes the portion of the video dataat a plurality of different bit rates so as to produce multiple versionsof the coded video data, (ii) generates a plurality of residual imagesfor each version of the coded video data, (iii) codes the plurality ofresidual images for each version of the coded video data using afine-granular scalability coding technique, and (iv) outputs one versionof the coded video data together with at least one corresponding codedresidual image therefor.
 30. An apparatus according to claim 29, whereinthe processor outputs the one version of the coded video data togetherwith at least one corresponding coded residual image therefor by (i)determining variations in a bandwidth of the receiver over time, and(ii) selecting which one of the multiple versions of the coded videodata and which of the coded residual images to output over time based onthe variations in the bandwidth of the receiver.
 31. Computer-executableprocess steps to code video data comprised of one or more frames, thecomputer-executable process steps being stored on a computer-readablemedium and comprising: a coding step to produce coded video data bycoding a portion of the video data using a frame-prediction codingtechnique; a generating step to generate residual images based on thevideo data and the coded video data; a coding step to produce codedresidual images by coding the residual images using a fine-granularscalability coding technique; and an outputting step to output the codedvideo data and at least one of the coded residual images to a receiver.32. An apparatus for coding video data comprised of one or more frames,the apparatus comprising: a first coding means for producing coded videodata by coding a portion of the video data using a frame-predictioncoding technique; a generating means for generating residual imagesbased on the video data and the coded video data; a second coding meansfor producing coded residual images by coding the residual images usinga fine-granular scalability coding technique; and an outputting meansfor outputting the coded video data and at least one of the codedresidual images to a receiver.
 33. A network system comprising: anencoder which receives input video data and which outputs frames ofcoded video data therefrom; a variable-bandwidth network over which theframes of coded video data are transmitted; a decoder which receives theframes of coded video data from the variable-bandwidth network and whichdecodes the coded video data; and a display which displays video datathat has been decoded by the decoder; wherein the encoder comprises: amemory which stores computer-executable process steps; and a processorwhich executes the process steps stored in the memory so as to producethe frames of coded video data by (i) coding a base layer from the inputvideo data using a frame-prediction coding technique, (ii) coding anenhancement layer from the input video data using a fine-granularscalability coding technique, (iii) determining a bandwidth of thevariable-bandwidth network, and (iv) selecting, for output, the baselayer and, in a case that the bandwidth of the variable-bandwidthnetwork is greater than a predetermined value, a portion of theenhancement layer.
 34. A network system according to claim 33, whereinthe predetermined value comprises a bandwidth that can accommodate thebase layer.
 35. A method of decoding coded video data comprised of anenhancement layer bitstream and a base layer bitstream, where the baselayer bitstream is coded using a frame-prediction coding technique andthe enhancement layer bitstream is encoded using a fine-granularscalability coding technique, the method comprising the steps of:receiving the coded video data; decoding the base layer bitstream usinga frame-prediction decoder; decoding the enhancement layer bitstreamusing a fine-granular scalability decoder; and combining decoded videodata from the base layer bitstream and from the enhancement layerbitstream to form a video image.
 36. Computer-executable process stepsstored on a computer-readable medium, the computer-executable processsteps to decode coded video data comprised of an enhancement layerbitstream and a base layer bitstream, where the base layer bitstream iscoded using a frame-prediction coding technique and the enhancementlayer bitstream is encoded using a fine-granular scalability codingtechnique, the computer-executable process steps comprising: a receivingstep to receive the coded video data; a decoding step to decode the baselayer bitstream using a frame-prediction decoder; a decoding step todecode the enhancement layer bitstream using a fine-granular scalabilitydecoder; and a combining step to combine decoded video data from thebase layer bitstream and from the enhancement layer bitstream to form avideo image.
 37. An apparatus for decoding coded video data comprised ofan enhancement layer bitstream and a base layer bitstream, where thebase layer bitstream is coded using a frame-prediction coding techniqueand the enhancement layer bitstream is encoded using a fine-granularscalability coding technique, the apparatus comprising: a memory whichstores computer-executable process steps; and a processor which executesthe process steps stored in the memory so as (i) to receive the codedvideo data, (ii) to decode the base layer bitstream using aframe-prediction decoder, (iii) to decode the enhancement layerbitstream using a fine-granular scalability decoder, and (iv) to combinedecoded video data from the base layer bitstream and from theenhancement layer bitstream to form a video image.
 38. An apparatusaccording to claim 37, wherein the frame-prediction decoder comprisesone of an MPEG-1 decoder, an MPEG-2 decoder, and an MPEG-4 decoder. 39.The computable-executable process steps stored on a computer-readablemedium according to claim 36, wherein the frame prediction decodercomprises one of an MPEG-1 decoder, an MPEG-2 decoder, and an MPEG-4decoder.
 40. A method for coding video data and for outputting codedvideo data to a plurality of receivers, the method comprising the stepsof: coding a first portion of the video data using a frame-predictioncoding technique to produce a first bitstream; coding a second portionof the video data using a fine-granular scalability coding technique toproduce a second bitstream; outputting the first bitstream to theplurality of receivers; dividing the second bitstream into two or moresub-streams; and outputting the two or more sub-streams to the pluralityof receivers.
 41. An apparatus for coding video data and for outputtingcoded video data to a plurality of receivers, the apparatus comprising:a memory which stores process steps; and a processor which executes theprocess steps stored in the memory so as (i) to code a first portion ofthe video data using a frame-prediction coding technique to produce afirst bitstream, (ii) to code a second portion of the video data using afine-granular scalability coding technique to produce a secondbitstream, (iii) to output the first bitstream to the plurality ofreceivers, (iv) to divide the second bitstream into two or moresub-streams, and (v) to output the two or more sub-streams to theplurality of receivers.
 42. Computer-executable process steps stored ona computer-readable medium, the computer-executable process steps tocode video data and to output coded video data to a plurality ofreceivers, the computer-executable process steps comprising: a codingstep to code a first portion of the video data using a frame-predictioncoding technique to produce a first bitstream; a coding step to code asecond portion of the video data using a fine-granular scalabilitycoding technique to produce a second bitstream; an outputting step tooutput the first bitstream to the plurality of receivers; a dividingstep to divide the second bitstream into two or more sub-streams; and anoutputting step to output the two or more sub-streams to the pluralityof receivers.